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Abstract 

Thirty-seven graduate students in two course sections were required to contribute to the online 
discussion of an instructional design case study. The discussion forums required different partici- 
pation protocols; students in section one implemented a constructive argumentation approach 
while students in section two had less structure for their postings. The results revealed that both 
sections exhibited co-construction of knowledge as measured using the content analysis protocol 
the Interaction Analysis Model ( IAM ), but the less structured section reached the highest phase 
of knowledge btiilding. Tn addition we found that the constructive argumentation approach 
and the task-oriented nature of the discussion influenced the interactions and co-construction 
of knowledge. The findings reveal practical implications for effective design of online discussion 
that may improve the quality of learning. (Keywords: computer mediated discussions, online 
discussion protocols, online discussion participation, argumentation.) 

INTRODUCTION 

With the proliferation of the Internet in our everyday lives, it is not surpris- 
ing that many universities and K-12 schools have adopted online or Web-based 
instruction as a delivery method. The National Center for Educational Statis- 
tics reported that in 1997-98, 58% of all post-secondary institutions offered 
Internet-based courses. Further, 82% reported that they had plans to increase 
their Internet-based offerings over the next three years (NCES, 1999), and the 
Sloan Foundation’s recent report indicates the enrollment growth rate for online 
courses far exceeds traditional higher education course growth trends (Sloan 
Foundation, 2004). Although education has progressed to a point where the 
Internet makes it easy for students to access online resources and tools, it is now 
important to focus on the online activities that facilitate students’ cognitive and 
metacognitive development. 

In many online courses, students are expected to share experiences, negotiate 
meanings, and construct knowledge within discussion forums. Vygotsky’s (1978) 
socio-cultural theory regards social interaction as an essential cognitive devel- 
opmental process (Pea, 1996; Salomon, 1988). Therefore, it is not unusual for 
instructors to require students to engage in collaborative learning activities, which 
requires social interaction. For discussions in face-to-face (FTF) courses, the in- 
structor can promptly redirect discussions to ensure that underlying assumptions 
are uncovered and that individual contributions do not ignore what others have 
said. However, this task becomes more difficult in online courses. If the discussion 
board contributions lack focus or the board content becomes confusing, then this 
critical component of an online course can both be an ineffective communication 
tool and actually impede learning. Hence, determining the appropriate discussion 
board protocols for participation may have a direct effect on the quality of learning. 
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A National Education Association poll of distance learning faculty suggests 
instructors believe that quality learning can occur in these Web-based courses 
(NEA, 2000), and further research has focused on the importance of the on- 
line discussion as a component of Web-based courses (Garrison, Anderson, & 
Archer, 2001; Harasim, 1989). To examine the quality of online discussions, 
one can implement content analysis methods to analyze discussion board con- 
tent (Mazur, 2004; Rourke, Anderson, Garrison, & Archer, 2001). Given the 
criticality of the online discussion forum to achieving such higher learning out- 
comes, we examined how having students engage in different discussion board 
participation protocols — one based on constructing formal arguments and the 
other not — affected knowledge construction in discussion forum postings. 

LITERATURE REVIEW 

The basis for our study comes from two literature areas: (a) online discussion 
forums as an important component of online courses, and (b) the use of argu- 
mentation as a means for promoting knowledge construction in online courses. 
We begin by examining the role of computer-mediated communication as op- 
erationalized by the online discussion forum in Web-based courses. 

Computer-Mediated Communication 

Computer-mediated communication (CMC) refers to asynchronous text- 
based communication that occurs through “one-to-one, one-to-many, e-mail- 
based discussion lists, bulletin boards” and computer conferencing environ- 
ment interactions between computer users (Romiszowski & Mason, 2004, p. 
397). CMC has emerged as an important research area in education, psychol- 
ogy, and technology domains (Hara, Bonk, & Angeli, 2000; Romiszowski 
& Mason, 2004). There are numerous studies that support the idea that 
interactions with the instructor and other students are essential elements in a 
Web-based course (Fulford & Zhang, 1993; Picciano, 1998; Sherry, 1996). In 
discussing online learning, Harasim (1989) describes interactivity as the most 
striking characteristic of CMC and the factor with the greatest potential to af- 
fect learning. Similarly, Garrison, Anderson, and Archer (2001) describe the 
importance in online learning of creating a “virtual community of inquiry” 
which allows learners to construct experiences and knowledge through analy- 
sis of the subject matter, questioning, and challenging assumptions. In a FTF 
environment, this kind of reflection is often accomplished via synchronous, in- 
teractive discussions and problem solving sessions. The ability to ask questions, 
share opinions, or disagree with the point of view in a reading assignment is 
vital to student learning. 

The online discussions can play a critical role in Web-based courses by helping 
learners construct knowledge (Jeong, 2003). In an effective online forum, the 
discussion encompasses the principles of constructivism and social interactions 
to help learners reach new insights, knowledge, and perspectives. Principles 
that demonstrate realistic tasks, scaffolds, and feedback for acquiring skills and 
knowledge, and collaboration with peers will facilitate the process of construct- 
ing a shared body of knowledge (Jonassen, 1999). These outcomes may be 


192 


Winter 2005: Volume 38 Number 2 
Copyright © 2005, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


best attained in the forum when appropriate scaffolding is provided within the 
students’ zone of proximal development (Zhu, 1998, as cited in Hara et ah, 
2000; Vygotsky, 1978). However, empirical evidence to indicate that text-based 
communication used in computer conferencing can facilitate higher-order and 
critical thinking is only just emerging and not entirely consistent in its results. 
For instance, Garrison, Anderson, and Archer (2000) report there is “limited” 
evidence that CMC can facilitate higher-order thinking. Other studies focus 
on learner-to-learner interactions in addition to the nature of those interactions 
(Collins & Collins, 1996; Ravits, 1997; Ward & Tiessen, 1997). These studies 
all reported that CMC can provide opportunities for more learner-to-learner 
interaction but also reported that CMC could facilitate more reflective and 
critical thinking skills. 

There is an earlier body of research that is more positive in its description 
of the potential effect of online forums on meaningful learning. These studies 
indicated that participation in online forums could lead to broader and deeper 
participation in group activities (Kiesler, Siefel, & McGuire, 1984; Pullinger, 
1986; Spitzer, 1989, as cited in Mazur, 2004). More recently, Jonassen and 
Kwon (2001) reported that during group problem-solving activities, the CMC 
participants produced fewer overall but more task-related messages than an FTF 
group. In addition, the CMC group’s decision-making patterns were more so- 
phisticated than that of the FTF group. 

Argumentation as a Means of Constructing Knowledge 

The basis for the treatment used in this study is founded in social construc- 
tivism and the potential knowledge construction benefits of having learners 
engage in argumentation. Constructivism refers to the belief that knowledge is 
“actively constructed by learners in response to interactions with environmental 
stimuli” (“Social Constructivism,” 2005). As a particular type of this cognitive 
constructivism, Vygotsky’s (1978) social constructivism is focused on the mean- 
ing making that occurs through the role of language and culture (Jonassen & 
Land, 2000; “Social Constructivism,” 2005). A general, albeit non-academic, 
definition of argumentation is a “heated discussion” or a debate that involves 
several individuals or parties (Andriessen, Baker, & Suthers, 2003, p. 5). A defi- 
nition that is aimed at argumentation in academic settings or settings intended 
to promote learning is based first on the term “argument” where an argument is 
a “meaningful expression” that is designed to support another utterance or posi- 
tion (Andriessen et ah, 2003, p. 6). Argumentation is then defined as a series of 
arguments that are linked by a common theme or topic. Arguments can be rhe- 
torical — i.e., one sided, where the argument is being used to tell others about a 
position and convince the ones receiving the argument of the relevance of the 
position. The transmissive quality of these “arguments” limits their capacity for 
a true multi-sided discussion that educators hope to promote in online discus- 
sion forums. Thus the argumentation activities we review refer to “dialogical” 
or multi-voiced arguments where different perspectives are being presented and 
examined and the goal of argumentation is to reach agreement on proposed 
claims (Driver, Newton, & Osborne, 2000). 
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Although the benefits of argumentation are not media specific, argumentation 
is recommended as a means to engage learners in a way that can increase think- 
ing deeply and meaningfully in an online environment (Derry, Levin, Osana, 
Jones, & Peterson, 2000). Andriessen et al. (2003) offer possible explanations 
for how argumentation can support such meaningful thinking. They describe 
argumentation as having the potential to support learning in three different 
ways: learning from the debate, learning about the debate, and learning to de- 
bate. They further propose that the possible explanations for bow students learn 
by engaging in argumentation may be based in several activities or processes 
that occur as part of the argumentation process. First, the actual act of produc- 
ing arguments or counter- arguments in an interactive context could help the 
producer to engage in reflection on his or her own position and possibly knowl- 
edge restructuring. Second, learning could occur in the form of the acquisition 
or release of viewpoints as a result of argumentation. Third, learners can engage 
in knowledge co-construction as a way of reaching compromise between dif- 
fering views. This phenomenon, which occurs around initial disagreement, 
can lead learners to hone meanings of concepts being used in the argument 
(Andriessen et al., 2003). 

Various techniques have been devised to support what is known as computer- 
supported collaborative argumentation (CSCA). Most are based on trying to 
provide participants with cues (graphic or text-based) on how to create the 
proper argument structure. One method of supporting argumentation develop- 
ment in online forums is through “constrained” discussion forums. 

Constrained or scaffolded discussion forums are pre-structured forms of con- 
versation systems that require participants to label each of their postings from 
a pre-defined set of message types. For example, Jonassen and Remidez (2005) 
constrained discussions about a problem by restricting the first-level responses 
to solutions. Responses to solutions were required to be one of the following: a 
reason to support, reason to reject, or modify proposal. Any responses to those 
warrants had to provide evidence, such as information or facts, personal opin- 
ion or belief, research findings, and so on. This is in contrast to the standard, 
non-constrained forum where users may label their messages with a subject 
heading but there are no enforceable rules that govern their participation. The 
rules that govern participation in a constrained forum are designed to encour- 
age certain types of thinking by the participants. The most commonly used 
constraint system supports argumentation, however, a limitless combination of 
rhetorical structures are possible. 

Constrained discussion can support different kinds of reasoning that in turn 
support more meaningful tasks. Cho and Jonassen (2002) found that using a 
constraint-based argumentation scaffold positively affected the ability of groups 
to collaboratively construct arguments in an online environment. Students 
using the conversation scaffold provided more comments related to problem 
definition, orientation comments, criteria development, solution development, 
solution approval, and solution critique. 

The potential effects of a constrained forum that implements rules to engage 
participants in argument construction can be achieved through the use of a regu- 
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lar (non-constrained) threaded discussion forum in conjunction with a set of rules 
defined by the facilitator. Jeong and Joung (2004) implemented such “manual” 
discussion constraints using message labels. One group was required to use a 
prescribed set of response categories (arguments, evidence, explanation, critique) 
as scaffolds for the content of their postings in online debates. Another group 
received the same categories (which were accompanied with explanatory defini- 
tions) to guide their posting content and were required to include a correspond- 
ing label in their message subject heading; a third group acted as a control. Unlike 
the graphical tool studies, Jeong and Joung’s (2004) study found that message la- 
beling reduced overall debate in the discussion and as a result reduced the propor- 
tion of arguments that were elaborated with explanations. Their study provides an 
important basis for understanding how commonly used threaded discussion tools 
may be “scaffolded” or “constrained” to affect effective participation in argumen- 
tation. The current study continues and expands this line of research. 

RESEARCH QUESTIONS 

The goal of this research is to compare evidence of knowledge construction 
from two discussion forums that addressed identical content but used different 
participation protocols. Consistent with an emerging qualitative methodology 
(Creswell, 1994), we address the following general research questions so as to 
not constrain our inquiry: What information, as measured by the LAM, is ex- 
changed among the students? To explore any differences between the participa- 
tion protocols, two sub-questions guided our study: 

• Did social construction of knowledge occur in both forums? 

• Did the participation protocols affect knowledge construction and 
participation? 

METHODOLOGY 

The study was implemented in an instructional design graduate-level course at 
a major Midwestern university in the United States. The course was a required 
component of an online masters curriculum for Educational Technology. Each 
author taught a course section using the same course materials. Students from 
both sections retrieved course content and assignments from a single course 
Web site; however, the two sections participated in separate asynchronous 
discussion board activities using Blackboard’s threaded discussion board tool. 
There were a total of 37 students enrolled in the online graduate course; 2 1 in 
section one and 16 in section two. In section one, there were nine men and 
fourteen women. Eighteen were pursuing a master’s degree and three were PhD 
students. Section two was composed of four men and twelve women. In section 
two, there were fourteen students pursuing a master’s degree, one PhD student, 
and one undergraduate. 

Instructional Context and Procedures 

The course was organized into fifteen weekly themes and topics. Discussion 
board participation was worth 5% of the final grade and consisted of weekly 
discussions on the assigned readings, instructional design topics, and case stud- 
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ies. For the case studies, participants were required to analyze and discuss in- 
structional problems that emphasized one or more aspects of the instructional 
design process (Ertmer & Quinn, 1999). The case study activity, which was 
worth 4% of the final grade, required each student to participate on a team that 
analyzed one assigned case, post three controversial case issues to the discussion 
board, and facilitate the online discussion for one week. As part of the required 
discussion board participation, the remaining students debated the lead team’s 
position on the controversial issues. 

Instructors for both sections (the researchers) provided identical case study 
guidelines (see Figure 1) that encouraged students to post substantive analysis 
and positions that were based on course readings, skills attained during the 
course, and/or personal research or experiences. The student-led nature of the 
case studies was designed to promote student knowledge construction and 
problem solving. Moreover, the cases themselves were intentionally authored to 
be ambiguous and open to multiple interpretations (Ertmer & Quinn, 1999). 
Given this, the objectives for the case study activity were for students to apply 
instructional design principles to each case by arguing for plausible case solu- 
tions rather than to acquire new instructional design content knowledge. To 
scaffold this process, each instructor facilitated a case study discussion in her 
own section before student-led cases began. Depending on the level of student 
engagement, the instructors posted messages to help students clarify their ideas 
and achieve a shared understanding regarding concepts. Although each section 
used the same content, teams identified and discussed issues that they deemed 
as important. We did not provide identical issues for each section nor did we 
prescribe when to scaffold, because we did not want to compromise the student 
analysis for the sake of research. 

Discussion Board Participation Protocols 

Protocols that define the rules and guidelines for discussion board participa- 
tion are important for establishing appropriate discussion board interactions. 
With protocols, students and facilitators will have common expectations for 
the quality and quantity of postings. Although the instructors encouraged 
students to participate in the weekly discussion board activities, they were 
required to participate in a minimum of seven weekly discussions. For par- 
ticipation in case study discussions, students in both sections were to discuss 
the controversial issues from the case posted by the instructor or a case study 
team. All students in both sections were instructed to post comments that 
were succinct, concise, and add value to the dialogue. Other common guide- 
lines for participation are shown in Figure 2. 

Students in section one followed the common guidelines outlined. Students 
in section two were instructed to follow guidelines designed to help them con- 
struct a formalized argument for supporting the positions they were proposing 
in their postings. Students’ postings were to be constructed according to the fol- 
lowing categories that are components of arguments (Toulmin, 1958). 

Thesis: A thesis is a claim or a proposition to be supported which deals with 
probability or facts. Every thesis should be followed with evidence for that thesis. 
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1 . Remember that there is no one right answer. As stated by Ertmer & 

Quinn (1999), “there are many answers... The solutions you propose will 
depend as much on the perspective you take as on the issues you identify” 
(p- 3). 

2. There is always more than one way to look at a situation. Pay attention to 
what others have to say (as presented in the case) and how that connects 
with your perspective. 

3. Be open-minded... and suspend judgment until you have considered all 
the ideas. “Be flexible and open-minded. Remember that problems can be 
attacked from many different angles” (Ertmer & Quinn, 1999, p. 4). This 
will be very important when you begin pondering the recommendation(s) 
and solution(s) with your Design Team. 

4. Try to avoid generalizations and extremes. Given the ill-structured nature 
of the situations in the case studies, it is likely that generalizations will not 
apply. Look to avoid words like always, never, everybody, or no one. 

Figure 1. Case Study Guidelines. 


1. Before creating a new thread or posting a comment in a forum, make sure that 
your specific topic or comment has not been posted on the board already. 

2. When replying to a posting, rename the subject area so that readers will 
quickly know your main topic. 

3. Although comments should be concise and succinct, comments such as “I 
agree” are not sufficient. 

4. You do not have to “dominate” the conversation to receive credit. Allow 
others to participate and contribute ideas. 

5. Only post comments related to the topic. Electronic mail or the other 
discussion forums should be used for comments to classmates that do not 
relate to the weekly topic. 

6. You must post comments to the guiding questions in the appropriate fo- 
rum or thread in order to receive credit. 

Figure 2. Rules for participating in the weekly discussions. 

Evidence: State a piece of evidence for or against a previously stated thesis. 

Assumption: State an underlying assumption either in the thesis put forth, or 
in response to another postings thesis or evidence statements. 

Synthesis: In this type of posting, one brings together or synthesizes prior 
postings. Such a posting may include a restatement of theses or evidence or as- 
sumptions, and perhaps offer an idea about the thesis that has the most merit, 
and why (e.g., more evidence). 

Students in section two were provided with definitions of each of these parts 
of an argument and were instructed to label each of their postings as one of the 
above listed types. 
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Data Analysis Method 

Our analysis methods are based on Chi’s (1997) eight-step method for “quan- 
tifying qualitative analysis of verbal data” (p. 271). She describes this method of 
mixing qualitative and quantitative methods as allowing researchers to use qual- 
itative coding methods to capture impressions and trends and then analyze the 
coded results quantitatively. Her process can be summarized as preparing the 
qualitative data by deciding on a unit of analysis, choosing or developing a cod- 
ing scheme and decide what will constitute evidence for each code, applying the 
scheme and depicting the results using graphs or tables (e.g., coding frequen- 
cies), and seeking a pattern or coherence in the data using quantitative methods 
and interpreting the results and their validity. This method allows researchers a 
means to systematically produce quantitative results from an underlying quali- 
tative coding of verbal or transcript data. Chi indicates that this methodology is 
most appropriate when the goal of the analysis is to “formulate an understand- 
ing of the representation of the knowledge used in cognitive performances and 
how that representation changes with learning” (p. 271). 

Our application of this method began by choosing to analyze the student-fa- 
cilitated discussion board content for four weeks where the two sections oper- 
ated using different participation protocols. For the weeks analyzed, students in 
section two had been instructed to use the “argumentation” format for postings 
and had in fact already applied these guidelines in prior weeks’ discussions. 

Our decision on the unit of analysis is described below and was directly 
tied to the choice of a coding scheme. Because we wished to gather evidence 
of knowledge construction in both groups, we chose to apply the Interaction 
Analysis Model (IAM) (Gunawardena, Lowe, & Anderson, 1997) to the discus- 
sion board transcripts. The IAM is based on a constructivist paradigm designed 
to detect evidence of knowledge construction, and was developed in an attempt 
to further understand and describe the processes of negotiating meaning and 
knowledge co-construction in a collaborative online discussion environment 
(Gunawardena et al., 1997) — both processes described by Andriessen et al. 
(2003) as being components of argumentation that may promote learning. 
Gunawardena et al. posited that when knowledge is being constructed, learners 
move through five phases (not necessarily sequentially). Table 1 provides a de- 
scription of the IAM phases. 

Using the IAM, results of coded transcripts are categorized into these five 
phases to provide a distribution that indicates the degree of knowledge con- 
struction activities. This distribution amongst the five phases provided the basis 
for Chi’s (1997) step of “depicting the results.” This model has since been used 
in studies by other researchers (Beaudrie, 2000; Kanuka & Anderson, 1998; 
Marra, Moore, & Klimczak, 2004; Schellens & Valcke, 2003). Given the lack 
of consistently adopted online content analysis models (Mazur, 2004), even this 
relatively small track record makes the IAM one of the most frequently used 
online content analysis models currently available. Not only is the IAM “well 
used,” it is also appropriate for our research questions because it detects knowl- 
edge construction; thus it is a good choice for our work. 
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Table 1: Interaction Analysis Model (IAM) (Gunawardena, Lowe, & 

Anderson, 1997) 


Phase 

Attributes 

Phase I: Sharing/comparing of infor- 
mation 

Statements of observation or opinion; 
statement of agreement between par- 
ticipants. 

Phase II: Discovery and exploration 
of dissonance or inconsistency among 
ideas, concepts, or statements 

Identifying areas of disagreement; ask- 
ing and answering questions to clarify 
disagreement. 

Phase 111: Negotiation of meaning/ 
knowledge co-construction 

Negotiating meaning of terms and 
negotiation of the relative weight to be 
used for various arguments. 

Phase IV: Testing and modification of 
proposed synthesis or co-construction 

Testing the proposed new knowledge 
against existing cognitive schema, per- 
sonal experience, or other sources. 

Phase V: Phrasing of agreement and 
applications of newly constructed 
meaning 

Summarizing agreement and meta- 
cognitive statements that show new 
knowledge construction. 


Each author separately coded the forum transcripts using the codes/phases as de- 
fined in the IAM protocol (see Table 1) — thus our decisions as to what utterances 
constituted evidence of each phase were based on the examples and explanations 
from the original IAM authors. To apply the IAM, the authors read each posting 
in the original sequence and applied an IAM phase or phases. In most cases, the 
authors coded multiple sentences, or a paragraph or two with a single phase; this use 
was consistent with the original application of the IAM (Gunawardena et al., 1997). 
Because many postings contained multiple paragraphs and addressed several topics, 
in many cases the authors assigned two or more phases to a single posting. 

To conduct inter-rater reliability checks, we adopted the convention based 
on prior research using this protocol (Beaudrie, 2000) to use the most ad- 
vanced phase from each posting as the basis for inter-rater checks. Hence, the 
unit of meaning became the entire posting, rather than phrases, sentences, or 
paragraphs within the posting. Given the developmental nature of this proto- 
col (Gunawardena et al., 1997) and that the highest levels of advancement of 
knowledge construction is directly implied by the most advanced phases, this 
convention seems both logical and viable. 

Although the IAM phases are clearly defined, applying the phases to the 
appropriate content required the authors to address the underlying meaning 
represented in the postings. Researchers have indicated that accounting for la- 
tent variables is somewhat subjective and depends on the rater’s interpretation, 
which is in contrast to identifying manifest variables that are easily observable 
(Rourke et al., 2001). For example, Phase III (see Table 1) uses “relative weight” 
as a variable for this phase, but it can have multiple indicators, such as “this is 
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important because...” or “I feel that it should be...” We addressed this poten- 
tial threat to reliability using our inter-rater checks during and after individual 
coding for our first data set (i.e., Week 6). To complete our application of the 
Chi (1997) methods, we conducted inter-rater coding checks for the remaining 
three data sets and organized the resulting frequencies into tables for interpreta- 
tion. These results are described in the next section. 

Another data source was summative student feedback regarding the course 
activities. During the last week of the semester, the students posted their evalu- 
ations of the effectiveness of their assignments and the weekly discussions — to 
provide a summative evaluation as well as formative data that would be applied 
to the next iteration of the course. 

RESULTS 

For the four case studies analyzed in both sections, the discussions included 
280 postings from students and instructors. Twenty-eight postings (10% of 
the total) were not coded because the posting in question was a lead team’s case 
solution, a student posting submitted to the wrong discussion thread, an exact 
duplicate of a prior posting, or an instructor posting describing participation 
guidelines. In all of these situations, the posting was not intended to contribute 
to knowledge construction, which was the focus of the current analysis. Of the 
remaining 252 postings, 177 postings were from section one and 72 postings 
were from section two. When examining the quantity of postings, there were 
obvious differences between the sections. The mean number of postings for case 
studies was 44.2 (SD= 9.63) for section one compared to 18 (SD=8.4) for sec- 
tion two. Table 2 provides a summary of the section differences. 

To compute inter-rater reliability figures, we followed the method delineated 
by Chi (1997), which requires that the raters categorize coding differences as 
either resolved, or in the instance of strong rating opinions by multiple raters, as 
unresolved. Table 3 reflects the resolved codings for section one. Week 6 was the 
first set of postings that we coded and it was the only week where the authors 
did not reach complete inter-rater agreement. The three postings where a strong 
disagreement remained amongst the authors were distributed amongst Phases 
I— III, which are also the three most frequently used phases (92%) for the agreed 
upon postings within each case discussion. KrippendorfPs (1980) reliability 
formula produced a coefficient of 61% for initial codes and 93% for codes af- 
ter reliability discussions. The Cohen’s kappa coefficient was also 61%, which 
represents a moderately strong agreement rate (Landis & Koch, 1977; Kvalseth, 
1989) based on three assumptions: (1) the units of analysis are independent, (2) 
the categories are mutually exclusive, and (3) the raters are operating indepen- 
dently (Cohen, I960) 1 . 

1 Although our inter-rater coefficient is on the lower edge of acceptability, these re- 
sults can be partially explained by the raters having to interpret another researcher’s 
category definitions. Many content analysis researchers develop their own coding 
scheme, which allows the researchers to have a complete understanding of how to 
apply codes. 
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Table 2: Discussion Board Participation Metrics by Section 


Participation Protocol 

Section 1 

Basic 

Section 2 
Constructive 
Argumentation 

Number of Students 

21 

16 

Case Studies 

Case Study 1 Postings (Week 4) 

31 

23 

Case Study 2 Postings (Week 6) 

45 

27 

Case Study 3 Postings (Week 12) 

47 

13 

Case Study 4 Postings (Week 13) 

54 

9 

Total Case Study Postings 

177 

72 

Mean Case Study Postings 

44.2 (SD=9.63) 

18 (SD=8.4) 

Mean number of students participating 

13 (-62%) 

10 (62%) 

Mean number of facilitators 

5 

3.75 

Mean number of student facilitator postings 

21.2 

4.2 

Mean number of Instructor/TA postings 

2 

3.2 


All Weekly Discussions 
Mean number of discussion board 
postings for case studies and other 

weekly topics 34.5 (SD=21.31) 16.9 (SD=10.96) 

Mean number of students participating 
per week 9 (46%) 8 (50%) 


Table 3: IAM Ratings for Section One, Basic Posting Requirements 



Phase I 

Phase II 

Phase III 

Phase IV 

Phase V 

Totals 

Week 4 

13 

12 

5 

0 

1 

31 

Week 6 

11 

16 

14 

4 

0 

*45 

Week 12 

23 

7 

16 

0 

1 

47 

Week 13 

20 

11 

18 

5 

0 

54 

Totals 

67 

46 

53 

9 

2 

177 


(.37%) 

(26%) 

(29%) 

(5%) 

(1%) 

(-98%) 


* Based, on total of 48 coded postings, including 3 where inter-rater agreement could not be 
reached. 


The results from section two also reflect Phases I — III as the three most fre- 
quently used phases for the agreed upon postings (see Table 4, page 202) with 
97% of coded postings falling in these three phases. The reliability coefficient 
for post-inter-rater reliability discussions was 100%. In contrast to section one, 
there were no postings coded at Phase V. 

We performed a Chi square analysis on the phase totals for each section. Due 
to low cell counts, we collapsed the Phases III — IV into one category because 
they all represent a type of knowledge construction. Chi-square analysis showed 
that the frequency differences between the two sections were significant (X* (2, 
N=249) = 7.11) at the p <.05 level. 
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Table 4: IAM Ratings for Section Two, Constructive Argumentation 

Requirements 

Phase I Phase II Phase III Phase IV Phase Y Totals 


Week 4 

13 

5 

5 

0 

0 

23 

Week 6 

9 

7 

9 

2 

0 

27 

Week 12 

11 

2 

0 

0 

0 

13 

Week 13 

7 

2 

0 

0 

0 

9 

Totals 

40 

16 

14 

2 

0 

72 


(56%) 

(22%) 

(19%) 

(3%) 


(100%) 


DISCUSSION 

Evidence of Knowledge Construction 

The distribution of our coded results in the IAM phases indicates that partici- 
pants in both sections were engaged predominantly in activities exploring agree- 
ment (Phase I), discovering or exploring dissonance or inconsistencies (Phase 
II), or engaging in the negotiation of meaning or knowledge co-construction 
(Phase III). These results are slightly different than prior research using this pro- 
tocol (Beaudrie, 2000; Kanuka & Anderson, 1998; Marra et al., 2004) where 
the preponderance of postings represented either Phase I or II. As defined by 
the LAM phases, knowledge construction and negotiation of meaning occurs at 
Phases III and above. Both sections showed evidence of knowledge construc- 
tion, with 29% of section one postings (see Table 3) and 19% of section two 
postings (see Table 4) occurring in Phase III. As noted previously, very few post- 
ings in either section were coded in Phases IV or V. 

Effect of Participation Protocols 

The effect of the differing discussion board participation protocols as mea- 
sured by evidence of knowledge construction can be examined from several 
different perspectives. We begin with examining the differences in all five IAM 
coded phases and then turn to the differences in the LAM knowledge construc- 
tion phases, and then conclude with a discussion of how participation quantity 
may have been affected by the participation protocols. 

Differences in IAM Coded Phases. The Chi-square analysis showed a signifi- 
cant relationship between the participation protocol and the LAM phases (X 2 
(2, N=249) = 7.11) at p < .05 with students in section one (the non-argumen- 
tation discussion board protocol) having both more coded postings and more 
postings at higher phase levels. Given the rather small frequencies, this finding 
is noteworthy. We discuss the possible explanations for these differences below 
and posit that they are related to an unexpected effect of the argumentation dis- 
cussion protocol. 

There are also strong similarities between the distributions of the coded phas- 
es for each section — particularly for the first three phases. In section one, a total 
of 166 (92%) coded postings were in Phases I— III. Similarly in section two, 70 
coded postings (97%) were in the first three phases. 

Differences in Knowledge Construction Phases. As seen in Tables 3 and 4, 
there was evidence of co-construction of knowledge for both sections, however 
Phase V was only evident in section one. This may be caused by the nature 
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of the discussion board task and/or the participation protocol. Regarding the 
discussion board task, having students respond to controversial issues defined 
by student facilitators regarding an instructional design case may not allow 
students to reflect upon and apply what they have learned because the discus- 
sion requires them to only focus on the presented issues. Students were asked to 
agree or disagree with the controversial statements. For both sections this may 
have inhibited students from moving beyond the “agree or disagree” phase to 
one where knowledge construction can occur and thus affected the distribution 
of Phases III— V ratings. 

A student indicated “I had a hard time working with a structured discussion 
board at first and didn’t get as much out of posting to another team’s case study 
as I did when we moderated the discussion ourselves.” This statement reflects 
two issues. One, students were constrained to the controversial issues presented 
by the student-led facilitators, and these in turn may have been affected by the 
content of the case. Two, the student-facilitators may have been the only ones 
to really construct and apply new knowledge. Because each case was assigned 
to student teams, the second issue is actually a positive outcome of the team as- 
signment. When students take more responsibility for leading discussion, they 
take more ownership of their learning. Similar to creating instruction, students 
were learning more when they were in charge of “leading” others or defending 
their analysis. 

The constructive argumentation protocol includes a “synthesis” type of post- 
ings, but this was not practiced by any of the participants in section two. Simi- 
lar to our research, Gunawardena et al.’s (1997) findings revealed that a debate 
format and the “task-oriented” nature of the discussion influenced the interac- 
tions and affected the co-construction of knowledge, both in a positive and 
negative manner. 

Participation Quantity. Even when one accounts for the differences in the 
section sizes (21 versus 16), there were clear differences between the number 
of postings and participation rates for the two sections. For student-facilitated 
case discussions, the average number of postings was 44 for section one and 1 8 
for section two. Over half of the students consistently participated during the 
case studies for section one, but there was a dramatic reduction toward the end 
of the semester for section two. Students in section two were active for the first 
two case studies ("81%), but only 56% and 31% participated for the third and 
fourth case, respectively. Recall that students in both sections were required to 
participate actively in seven forums to earn full credit for the discussion forum 
part of the course. By the time cases three and four were discussed (weeks 12 
and 13 of the 15-week semester), many students may have met this requirement 
and thus may have focused on the large, high-point-value instructional design 
projects that they were completing at that time. However plausible and practi- 
cal, this situation was true for both sections one and two and thus does not 
account for the differences in participation between the two sections. Thus, we 
turn to other explanations that may account for these differences in participa- 
tion quantity - and some of which may also explain the differences in participa- 
tion quality described previously. 
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The nature of the discussion board protocol for section two is an obvious pos- 
sible explanation for both the participation quantity and quality. The argumen- 
tation protocol used in section two required students to formulate their post- 
ings in one of five types (thesis, evidence for, evidence against, assumption, and 
synthesis), where each type mapped to a part of a formal argument. In her writ- 
ten feedback about the course at the end of the semester, one student described 
how the format affected her participation: 

I enjoyed the case studies — I seemed to get hung up on the correct 
method of discussion sometimes, instead of focusing on the meat of 
the case. I might have submitted more if I were more secure in the 
proper method of submission. The case studies seemed to make the 
meaning of instructional design more clear. 

Although students may have experience with online courses and participat- 
ing in online forums, it appeared that they did not have experience with con- 
structing an argument. Even though the section two instructor implemented a 
“practice” discussion with students using the protocol and also provided facili- 
tative feedback on their use of the protocol, students may still have found the 
required protocol difficult to implement. Additionally, although argumentation 
is an essential skill for many problem-solving activities (Jonassen & Remedez, 
2005), the argumentation protocol may not have been well suited for the case 
analysis task. Evidence for this explanation is that there were fewer postings in 
section two discussions. Furthermore, the authors noted during their analysis 
that section one postings often contained several different ideas associated with 
the controversial issue the student was discussing. For instance, in the follow- 
ing excerpts from a Phase III posting from the fourth case study, this section 
one student addresses the role of the instructional designer relative to the client 
and subject matter experts (paragraphs 1 and 3), the poster’s relevant personal 
experience on designing curriculum (paragraph 2), and the issue of platform 
compatibility that was a key aspect of the case (paragraph 4). 

Clients depend upon the instructional system designer to provide 
an educational or training product that not only meets their needs 
“now” but at least for the foreseeable future. In addition, there are a 
number of aspects of system design about which subject matter experts 
have little knowledge and therefore depend upon the instructional sys- 
tem designer for expertise. 

I recently had occasion to provide consultation on a project in 
which an entire curriculum of 8 courses had been created. For these 
online courses, all lectures were in digital audio format — no text alter- 
native. Of course, this format explicitly restricted access to the lectures 
by those students with a hearing impairment. 

Regardless of one’s perspective concerning a given platform, I ar- 
gue that it is incumbent upon the instructional system designer to 
understand the broader implications of the final instructional product 
and the constraints imposed by institutional policy, the law, technol- 

204 Winter 2005: Volume 38 Number 2 

Copyright © 2005, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


ogy, and the Zeitgeist — and to bring this knowledge to bear on a given 
instructional product. Otherwise, it seems to me, we are just program- 
mers and frankly — what’s the use? 

By the way, 5 years ago Apple Computer was in the worst financial 
condition in its history having lost some $68 million in the last quarter 
of 1996. It wasn’t until 2 years later (1997 approximately) that the com- 
pany began its current upswing . . . Surely, Nguyen was aware of this 
fact when she considered the development of the instructional product. 

If a student in section two had wished to post similar ideas, that student 
would have needed to determine how to “parse” all of these individual ideas 
into the required message types of the constructive argumentation protocol. 

This extra effort may have led students in section two to be somewhat stifled in 
their postings — which could in turn lead to fewer postings, lower overall par- 
ticipation rates, and potentially lower motivation to participate in the forums. 
The last idea is supported by the student’s comment that “I might have submit- 
ted more if I were more secure in the proper method of submission.” Further 
support for this explanation comes from prior work from Jeong and Joung 
(2004). The researchers found that when students participating in an online 
forum were required to pre-classify their messages with response headings, those 
students participated in the forum to a lesser degree than the control group and 
also showed less evidence of engaging in argumentation — which was the intent 
of the message labeling activity. 

Other explanations not directly linked to the protocol are also possible. For 
instance, one may propose that the actual content of the different cases may 
have affected both the quantity and quality of the participation we observed; 
however, given the nature of the differences we saw between the sections in each 
individual case, this seems a less likely explanation. Another explanation could 
be that some students were better facilitators than others, which can affect the 
quantity and quality of postings. The challenge for facilitators is to demonstrate 
appropriate, relevant, and timely responses that bring attention to important 
ideas and make linkages to other postings (Garrison & Anderson, 2003). 
During our analysis, we coded facilitator postings as Phase ID, because they 
reflected probing questions that clarified or directed the discussion. Our results 
indicate a considerable difference in the number of student facilitator postings. 
(See Table 2.) We recognize that a facilitator analysis would provide a different 
lens for examining the data and could contribute to understanding the quantity 
and quality of postings; however the focus of our analysis was the evidence of 
knowledge constructing and not the facilitator methods for encouraging dis- 
course among the students. 

The lower enrollment in section two may have contributed to the overall low- 
er participation rates. Excluding the student facilitation team, there were only 
a maximum of 12 students to debate the case’s controversial issues. With this 
relatively small number of students distributing their postings over three differ- 
ent controversial issues, this may have resulted in both less in-depth analysis of 
any single issue as well as an overall lower number of postings and lower partici- 
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pation rates. In short, with so few students, it may have been difficult to build 
up the needed momentum to hold an active and engaging discussion. This 
potentially affected participation rates and the degree to which knowledge con- 
struction occurred in section two. If a smaller number of students’ participation 
is distributed over the three or more controversial issues posted for each case, 
it may have been less likely that the discussion of any one of the controversial 
issues would build enough for the students to engage in meaning making and 
knowledge co-construction activities (e.g., LAM phases III and higher). 

This problem may have been exacerbated in the last two cases, when some 
student may have reached their required discussion participation. Once the sev- 
en weekly participations were met, students did not feel obligated to participate 
in discussions. In section one, this effect may not have been so pronounced, as 
there were more students that could have spread their participation throughout 
the semester. This is particularly plausible because of the “non-redundancy” 
requirement. (See Figure 1 .) Section one students may not have had the op- 
portunity to post at the beginning because they waited too late and most of the 
appropriate and relevant ideas had already been posted. 

The question of how often students should participate depends on the objective 
of the activity and in this case, the number of students. One could implement a 
strategy to separate students into smaller groups and discuss the same topic, but 
this requires more management for the instructor and students in each group 
might miss out on important ideas from the other group. In their end-of-semester 
comments, many students indicated that they liked the ability to choose when to 
participate. One student commented about not having to post every week: 

I for one appreciate your view of the discussion board as a support for 
the class. Like Bob, I am also enrolled in another class that demands a 
great number of postings to the discussion board each week. While I 
enjoy reading others’ views and feel that the interaction is very valu- 
able, I want to post when I feel that I have something to add to the 
discussion or have a question. I don’t like having to post just to meet a 
quota each week. 

Another student noted that decreasing the amount of participation meant 
“less fluff and less clicking on meaningless post-just-to-submit posts.” Research 
from Bodzin and Park (2000) supports this explanation, as they found the pos- 
sibility of a saturation effect when a large number of students are required to 
respond to a single problem. With 32 students responding in one week, Bodzin 
and Park (2000) discovered that many students were not reading other postings, 
and were posting to meet the requirements. The implications of our findings 
and other research support participation strategies that allow flexibility for stu- 
dents to choose when to contribute to discussions. 

CONCLUSIONS AND RECOMMENDATIONS 

This study examined the effect of differing discussion board participation pro- 
tocols on evidence of knowledge construction as measured by the LAM using 
following research questions: 
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• Did social construction of knowledge occur in both forums? 

• Did the participation protocols affect knowledge construction and 
participation? 

The argumentation protocol that was designed to facilitate knowledge con- 
struction in section two did not show evidence of having that effect. In fact, 
our findings show that there were overall fewer postings in section two, a lower 
number of postings in Phases III and above, and that the overall level of in- 
teractions in this section was lower than in section one. Our discussion notes 
that some of these differences may be due to the smaller number of students in 
section two, but student comments and prior research (Jeong & Joung, 2004) 
indicate that the argumentation protocol, as we implemented it, may have 
negatively affected students’ quantity and quality of participation. These results 
suggest the following implications for researchers and instructors for the discus- 
sion forums in online courses. 

Discussion Board Tasks and Participation 

When instructors are creating discussion board activities for online courses, at 
least two questions must be answered. First, what is the objective of the discus- 
sions? Different objectives might be to create a “social presence” among students 
so that they do not feel isolated, to ask questions regarding assignments or top- 
ics, or to determine if students understand a topic by having them analyze and 
evaluate contextual situations. Based on the response to this question, different 
rules might be implemented to focus on the quality of the interaction more so 
than the quantity. The second question is, how important is online discussions 
in comparison to the other activities that students will perform? This question 
alludes to the amount of participation that instructors expect from students in 
online discussions along with the other required activities for the course. If a 
small percentage of student effort is designated for class participation, our re- 
sults show that it can affect the quality and quantity of interactions. 

For our course, the minimum participation quantity affected the number of 
people that contributed each week. This can be viewed as positive and nega- 
tive. For students that did not really have anything to contribute, they did not 
post redundant or disjointed comments to the discussion board in order to 
meet the requirements. On the other hand, if there is not 100% class partici- 
pation, then it might lessen the co-construction of knowledge: someone may 
be able to contribute, but will not because he/she has met the requirements 
and there is a greater need to focus on other activities. Therefore, we recognize 
that it is good to have options when it does not decrease the active partici- 
pation. This becomes even more critical when you have online classes with 
small enrollments, such as less than 15 students. When it is more than 15, 
this strategy would probably be more effective. By balancing the number of 
required discussion participations during the semester with the other course 
activities, students might be able to provide more meaningful and “thought- 
provoking” discussions when they do participate. Thus, instructors need to 
determine how often students must participate in order to meet the learning 
objectives. 

Journal of Research on Technology in Education 207 

Copyright © 2005, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


Regarding the facilitation of the discussions, some student facilitators were 
more aggressive in moving the discussion to a consensus than others. Facilitator 
efforts may have been hampered by requirements to post all of the issues at the 
beginning of the discussion. To have more evidence of synthesis and application 
of newly constructed meaning (i.e., Phases IV and V), we believe that focus- 
ing on one issue at a time rather than introducing another one will assist with 
reaching higher phases. Another strategy would be to impose a time limit for 
each issue, such as two days, to improve timely involvement. As many discus- 
sion board participants have experienced, it can be difficult to get reacquainted 
with a discussion after several days have passed. Thus, a different time period for 
each issue may contribute to more meaningful discussions. Research examining 
variables for time lengths, day of the week, and number of participants along 
with facilitation methods might provide strategies for increasing participation 
without compromising the quality of participation. 

Participation Protocols 

Further research is needed to investigate the effect of participation protocols 
with different discussion board activities. Our findings are limited by the small 
class size for section two in comparison to section one. Also, the investigation 
of participation protocols needs to include interventions that are more “struc- 
tured” for requiring students to use certain types of postings at different stages 
of the discussion. Our argumentation approach was a self-imposed structure 
and the course management system (Blackboard) offered no support for the 
protocol. However, researchers and practitioners now have access to discussion 
board software packages that directly support instructor-defined protocols for 
discussion forums, including CaMILE (Guzdial, 1995), Belvedere (Suthers, & 
Hundhausen, 2001), and CSILE (Scardamalia & Bereiter, 1994). Thus, further 
research using participation protocols implemented in these packages may pro- 
vide students with an externally defined structure that will facilitate their better 
use of the protocol. 

Lastly, our results indicate that in designing discussion board activities and 
protocols for participation, instructors must try to ensure that the participation 
protocol is in alignment with the task and intended objectives of the discussion 
board task. Although argumentation has been shown to facilitate higher-order 
thinking and learning outcomes (Andriessen, et. al. 2003; Jonassen & Remedez, 
2005), applying an argumentation protocol to the task of discussing contro- 
versial issues in the instructional design case studies may have been a mismatch 
between the requirements of the case study task (agreeing or disagreeing with 
controversial issues) and the cognitive skills necessary to build a meaningful 
argument. Software tools for implementing discussion boards with externally 
defined participation protocols, or constrained discussion forums, can allow 
researchers to examine the relationships between different types of participation 
protocols and discussion board tasks and accompanying learning outcomes that 
produce the most effective learning. Additionally, further research such as the 
current study using more commonly available discussion tools may also fur- 
ther the communities’ understanding of how to promote meaningful learning 
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through online forums. Although our results indicate that our case study task 
did not mesh will with an argumentation protocol, one should not conclude 
that argumentation may not promote knowledge construction when paired 
with a different discussion board task. 
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